Skip to content

fix(eval): make the synthetic fundamentals gate real — post-#201 review - #202

Merged
slittycode merged 1 commit into
mainfrom
claude/asa-integration-producer-bp82ws
Jul 3, 2026
Merged

fix(eval): make the synthetic fundamentals gate real — post-#201 review#202
slittycode merged 1 commit into
mainfrom
claude/asa-integration-producer-bp82ws

Conversation

@slittycode

Copy link
Copy Markdown
Owner

Review-and-fix of #201 (Codex's PR-A1 implementation of the accuracy program). The structure was right, but neither committed gate had ever been run: asa verify's backend leg was red on main (3 failing default-manifest checks) and the committed synthetic manifest failed 52 of 140 checks when executed. CI never caught it because the workflow only runs unittest discover.

What was broken (all verified by running the gates)

  • drum_stem_known_counts: snare detector counted the 16 kicks (kick click/pitch-sweep energy lands in the 120–2000 Hz band, and Codex placed snares on kick beats so coincident onsets merge); hat detector found 4–8 of 32 hats (below its floor). Counts read 15 and 8 vs truths 8 and 32.
  • four_on_floor_clear_128: BPM measured exactly right (128.0, Percival 128.4, agreement) yet failed quality:tempo — 4-bar clips cap the extractor's confidence below the 0.7 bar.
  • Synthetic manifest: mono_bass_transcription missing --transcribe (noteF1 = 0), chord fixtures spelled sharps against the Viterbi's flat vocab (F# major scored 0.25), key comparison lacked enharmonic folding (Db minorC# Minor), BPM checked on transient-free chord clips (read 61–98 vs 96–140 truths), spec feat(ui): re-analyze CTA for legacy Session Musician runs #29 silently dropped by a [:28] slice, ground-truth-for-later (hitTimes, swingPercent) embedded inside expected.

Fixes

Generator — engineered band-disjoint one-shots (kick = 50 Hz sine with a load-bearing 20 ms fade-in and 0.20 s length; snare/hat = double-filtered noise bursts); grid clips are a bare accented-kick pulse (off-beat hats bin ambiguously in analyze_time_signature's ±half-beat counting and flipped 4/4 reads to 3/4 or 6/8); swing clips keep swung hats but check BPM only; chord labels computed from pitch classes, flat-spelled; per-kind expected blocks with everything else under truth; --transcribe on the bass fixture; all 29 specs render; order-independent determinism.

Harness — enharmonic pitch-class folding for key + chord label comparison; knownGaps: per-track check names that still run and report scores but don't gate (checksInformational in the summary), keeping measured baseline weaknesses visible as improvement targets.

Trust layerbpmAgreement=True with confidence ≥ 0.5 is now authoritative: a BPM measured exactly right and cross-confirmed by an independent estimator was being hedged as "cross-check is not strong enough", which was factually wrong. Covered by new tests; golden unaffected.

CI — the default-manifest fundamentals gate (render + evaluate_fundamentals.py --fail-on-skip) now runs in the backend job, closing the hole that let #201 merge with a red gate.

Results

  • Default manifest (the asa verify / CI path): 15/15 checks green.
  • Synthetic manifest: 29 clips, 54/54 active checks green, 0 failed, 8 informational knownGaps (odd-meter detection + downstream downbeats, tempo octave at 174 BPM, 7/8 tempo wobble) — quantified in audits/accuracy-baseline-2026-07-03.md, the accuracy program's "before" column.
  • Notable baseline finding: the beat grid is strong everywhere (F1 0.95–0.98 even in 3/4, 6/8, 7/8); meter → downbeats is the weak layer — exactly the case for running the pre-registered beat_this gate next.
  • Backend suite: 1283 tests OK. Both gate commands exit 0 and reproduce deterministically.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL


Generated by Claude Code

Review of #201 found the committed gates had never been run: 'asa verify'
was red on main (3 failing default-manifest checks) and the synthetic
manifest failed 52 of 140 checks. This commit makes both gates green with
honest measurements and records the first accuracy baseline.

Generator (build_synthetic_corpus.py):
- Engineered band-disjoint one-shots replace product drum samples in
  fixtures: the kick's pitch-sweep onset/click landed in the snare band
  (16 kicks counted as snares) and product hats never cleared the hat
  detector's floor. Kick = 50 Hz sine, 20 ms fade-in (sharper attacks
  splatter into the snare band), 0.20 s length (its low-band tail
  otherwise turns -64 dB snare residue into countable local maxima);
  snare/hat = band-passed bursts with an extra HP pass.
- Grid clips are a bare accented-kick pulse: 'and' hats bin ambiguously
  in analyze_time_signature's ±half-beat onset counting and flipped 4/4
  reads to 3/4 or 6/8. Swing clips keep swung hats but check BPM only.
- Chord labels computed from pitch classes, flat-spelled to match the
  Viterbi vocab (hardcoded sharp table scored 0.25 on F# major).
- Bass fixture gets --transcribe (was missing → guaranteed noteF1 0).
- expected blocks carry active checks only; ground-truth-for-later
  (swing %, hit times) moves to a separate 'truth' key.
- Dropped the [:28] truncation that silently discarded multi_F_major;
  one-shots carry their own seeds (order-independent determinism).

Harness (fundamentals_evaluation.py):
- Enharmonic pitch-class folding for key + chord comparisons
  ('C# Minor' == 'Db minor', 'Gb' == 'F#'); relative-key table replaced
  with pitch-class math.
- knownGaps: per-track check names that still run and report scores but
  don't gate — measured baseline weaknesses stay visible as improvement
  targets (8 at baseline: odd-meter detection + downstream downbeats,
  tempo octave at 174, 7/8 tempo wobble).

Trust layer (fundamentals_quality.py):
- bpmAgreement=True with confidence >= 0.5 is authoritative: a BPM
  measured exactly right and cross-confirmed by Percival was being
  hedged as 'cross-check not strong enough'.

CI: the default-manifest fundamentals gate (render + --fail-on-skip) now
runs in the backend job — #201 merged green because CI never executed
the gate it shipped.

Baseline: audits/accuracy-baseline-2026-07-03.md — 29 clips, 54/54
active checks green, 8 knownGaps quantified. Backend suite: 1283 tests OK.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL
@slittycode
slittycode merged commit c5c84e0 into main Jul 3, 2026
4 checks passed
@slittycode
slittycode deleted the claude/asa-integration-producer-bp82ws branch July 3, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants